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ABSTRACT 

’  V- 

The  Customer  Feedback  Office  of  the  Product  Assurance  Directorate,  MICOM  has 
the  mission  to  manage  and  analyze  data  in  the  Deficiency  Reporting  System  and  make 
recommendations  for  appropriate  actions.  This  report  presents  the  results  of  a  contract 
delivery  order  for  support  to  the  CFO  performed  by  the  Product  Assurance  Division  of 
Hilton  Systems,  Inc. 
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The  contractor  was  issued  delivery  order  number  20  as  an  expansion  of  delivery 
order  number  5.  Delivery  Order  5  was  concerned  with  improving  the  present  day-to-day 
operations  of  CFO,  while  the  tasks  discussed  in  this  report  investigate  the  application  of 
certain  additional  capabilities. 

The  contractor  was  directed  to  recommend  ways  of  establishing  statistical  control  of 
the  time  required  to  initiate  an  investigation.  In  addition,  the  contractor  was  to  develop  a 
plan  for  the  implementation  of  a  predictive  analysis  capability  within  CFO. 

The  discussion  contained  in  this  report  is  limited  to  these  tasks,  i.e.  Statistical 
Process  Control  and  Predictive  Analysis.  For  completeness,  the  discussion  of  work  directly 
related  to  Delivery  Order  5  is  discussed  in  the  Final  Technical  Report  for  that  delivery 
order. 

Progress  made  in  meeting  these  requirements  is  discussed  herein  and  this  report 
follows  the  format  of  data  item  description,  UDI-S-23272C. 
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2.0  Discussion 


2.1  Statistical  Process  Control 

The  contractor  was  directed  to  analyze  the  response  times  associated  with 
completing  Quality  Deficiency  Report  (QDR)  actions  from  receipt  to  closing  with  the  goal 
to  establish  statistical  process  controls.  The  specific  subprocess  to  be  reviewed  was  the 
time  required  to  initiate  an  investigation  on  a  QDR. 

The  analysis  started  with  a  review  of  all  the  pertinent  regulations  and  PAD  draft 
SOP  No.  702-QA-l,  Processing  of  Quality  Deficiency  Reports  (QDRs).  A  number  of 
specific  time  periods  were  stated  in  the  documents  reviewed,  but  none  set  a  limit  for  when 
an  investigation  letter  had  to  be  completed.  There  was  mention  of  a  1984  PAD  SOP  that 
stated  this  time  to  be  NLT  7  days  after  receipt  of  a  QDR,  but  the  analyst  was  unable  to 
verify  this. 

With  this  background,  statistical  control  charts  were  developed  by  system  and  by 
action  officer  for  the  first  six  months  of  FY89.  This  provided  a  snapshot  look  at  the  time  it 
now  takes  for  the  needed  investigations  to  be  initiated.  This  established  a  bench  mark 
from  which  the  PAD  managers  can  make  a  decision  as  to  what  limit,  if  any,  they  wish  to  set. 

The  following  action  processing  codes  were  selected  as  being  those  that  initiate  an 
investigation  of  some  sort: 

A  -  Transferred  to  a  new  screening  point  within  the  Army 

M  -  Action  requested  from  overhaul  facility 

P  -  Screen  supply  system 

X  -  Forwarded  to  participating  component  for  action 

Y  -  Received  from  participating  component  for  investigation 

2  -  Investigation  by  other  than  Army 

4  -  Action  requested  from  Mat_Mgt. 

5  -  Action  requested  from  Proc_Pdn 

6  -  Action  requested  from  R&D 

7  -  Action  requested  from  other  local  office 

8  -  Sent  exhibit  for  evaluation 
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In  all  cases,  the  first  occurrence  of  any  of  the  above  codes  was  considered  the  start  of  an 
investigation.  For  several  QDRs  there  were  multiple  entries.  Appendix  A  provides  300 
data  points  which  were  subgrouped  into  sets  of  10  for  analysis.  For  each  group  of  10  action 
code  days  the  average  and  the  range  were  calculated.  After  each  group  average  and  range 
were  determined,  overall  average  and  the  average  of  all  the  ranges  were  calculated.  This 
provided  an  estimate  of  the  grand  average  and  average  range  for  the  total  population  of 
QDRs  requiring  an  investigation  action. 

Using  the  Shewhart  concept  of  variation,  upper  and  lower  control  limits  were  set  at 
a  distance  of  three  standard  deviations  on  either  side  of  the  estimated  averages.  The 
constants  contained  in  Table  1  below  are  used  to  simplify  the  arithmetic  in  the  following 
formulas: 

UCLx  =  X  +  A2  R 
LCLx  A2R 
UCLr  =  D4R 


LCLr  =  D3R 
Where: 

UCL  =  Upper  Control  Limit 
LCL  =  Lower  Control  Limit 

X  =  mean  or  average  of  each  subgroup  of  10  data  elements 
X  =  the  average  of  the  10  group  averages 
R  =  range 

R  =  the  average  of  the  ranges 


TABLE  1 


CONTROL  CHART  CONSTANTS  FOR  AVERAGE  AND  RANGE  CHARTS 
BASED  ON  THE  AVERAGE  RANGE 


SUBGROUP 


SIZE 

Ao 

d3 

Da 

2 

1.880 

0 

3.268 

3 

1.023 

0 

2.574 

4 

0.729 

0 

2.282 

5 

0.577 

0 

2.114 

6 

0.483 

0 

2.004 

7 

0.419 

0.076 

1.924  _ 

8 

0373 

0.136 

1.864 

9 

0.337 

0.184 

1.816 

10 

0.308 

0.223 

1.777 
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Using  a  subgroup  size  of  10  the  following  values  from  Table  1  are  placed  in  the  above 
formulas:  A2  =  0.308;  D3  =  0.223;  D4  =  1.777.  The  average  mean  (?)  time  to  start  an 
investigation  at  present  time  was  13.9  days,  with  a  calculated  upper  limit  of  24.5  days  and  a 
lower  limit  of  3.3  days.  The  average  range  (R)  is  34.3  days,  with  an  upper  limit  of  60.9  days. 
As  can  be  seen  from  Appendix  B  there  were  2  incidents  of  upper  bound  violation  in  mean 
and  5  incidents  in  range. 

Appendix  C  is  a  presentation  of  the  same  data  from  a  monthly  standpoint.  The  data 
points  were  selected  by  action  process  date  and  placed  in  subgroups  of  five  within  each 
month.  To  compensate  for  the  dropping  of  subgroups  of  less  than  five,  the  month  of  April 
was  added  to  the  population.  In  order  to  provide  a  month  to  month  comparison,  the 
population  average,  range  and  control  limits  were  used  to  plot  the  monthly  data. 

The  same  procedures  discussed  above  were  used  with  the  values  for  A2  ,  D3  and  D4 
taken  from  the  subgroup  size  5. 

As  can  be  seen  on  the  charts  in  Appendix  C,  the  average  mean  is  14  days,  with  an 
upper  limit  of  27.7  days  and  an  average  range  of  23  days,  with  an  upper  limit  of  48.6  days. 
Interesting  to  note  is  that  by  this  method,  with  data  regrouped  by  fives,  we  now  have  4 
violations  of  upper  limit  in  mean  (October,  December,  January,  February)  and  7  in  range 
(October,  November,  February)  verses  the  2  and  5  respectively  seen  on  the  chart  in 
Appendix  B. 

Next  we  show  the  data  by  action  officer.  The  computer  was  programmed  to  sort  all 
QDRs  by  action  officer  code  and  subtract  the  date  an  investigation  code  appeared  from  the 
received  date.  This  gave  the  analyst  a  list  of  all  QDRs  by  action  officer  with  the  number  of 
days  required  to  take  action  listed  (Appendix  D).  These  points  were  then  grouped  by  five 
and  the  charts  in  Appendix  E  formulated.  The  average,  range  and  control  limits  were 
computed,  as  stated  earlier,  for  each  action  officer.  This  gave  a  picture  of  how  consistently 
each  one  performed. 

Table  2  is  a  matrix  of  all  action  officers  and  the  population  depicting  how  each 
performs  individually  against  the  group  averages.  These  numbers  demonstrated  one 
possible  method  that  a  manager  could  use  in  evaluating  the  performance  of  his 
organization  both  collectively  ana  individually. 

TABLE  2 


CONTROL  CHART  DATA  COMPARISON 

TOTAL  #  ODRS 

X 

UCUX) 

R 

UCUR) 

POPULATION 
ACTION  OFFICER 

352 

14 

27.7 

23  . 

48.6 

GSR 

145 

12.8 

20.0 

12.5 

26.4 

JC 

60 

29.0 

57.8 

50.0 

105.7 

LMB 

95 

9.2 

22.0 

22.0 

48.0 

SEW 

40 

9.9 

18.0 

14.0 

29.0 

SJS 

12 

15.6 

20.0 

4.0 

10.0 
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All  the  above  is  for  demonstration  of  principle  purposes.  The  data  used  was  taken 
from  a  prior,  11  August  1989,  download  of  data  from  the  Information  Management 
Directorate  (IMD)  and  is  only  representative  of  the  total  QDR  database. 

2.2  Predictive  Analysis 

The  ability  to  predict  what  will  happen,  with  a  high  degree  of  accuracy,  to  a  weapon 
system  or  repair  part  once  it  has  been  fielded  has  long  been  the  goal  of  development  and 
support  personnel.  Predictive  analysis  techniques  were  examined  for  the  purpose  of 
providing  such  a  capability  within  PAD. 

The  plan  attached  at  Appendix  F  presents  a  sequential  methodology  to  achieve  this 
goal.  The  following  paragraphs  provide  a  discussion  of  a  system  being  implemented  at 
TACOM.  This  system,  with  modification,  may  have  application  for  MICOM  and  moves 
the  process  to  step  3  of  the  proposed  plan. 

The  Fielded  Vehicle  Performance  Data  System  (FVPDS)  integrates  data  from  a 
number  of  sources  into  a  single  database  and  is  being  evaluated  for  use  as  a  possible  AMC 
standard  system.  The  program  runs  on  a  mainframe  computer  and  requires  5.7  gigabytes 
of  direct  access  storage.  The  data  management  program  used  by  FvPDS  is  an  AMC 
system  called  Model  204,  which  is  also  being  considered  as  an  AMC  standard.  FVPDS 
contains  2  years  of  historical  data  which  may  or  may  not  be  acceptable  to  MICOM.  It  has 
been  designed  to  use  the  source  systems  data  as  it  comes  in  ana  to  make  comparisons  for 
trend  and  predictive  analysis  purposes. 

TACOM  plans  include  the  establishment  of  a  functional  control  group  of  6 
personnel  to  control,  operate  and  maintain  the  system  as  well  as  the  equivalent  of  2.6 
personnel  at  installation  level  to  maintain  and  operate  the  mainframe  portion.  This  is  a 
total  of  8.6  full  time  personnel  for  FVPDS. 

The  system  is  menu  driven,  considered  user  friendly  and  simple  to  use.  It  has  four 
levels  of  access  security  so  that  users  will  only  be  able  to  access  that  for  which  they  have  a 
need.  Access  control  will  be  maintained  by  the  functional  control  group.  Working  access  is 
desired  for  all  personnel  of  TACOM  and  this  is  a  matter  of  availability  of  funds  for 
terminals  and  training.  This  wide  dissemination  is  possible  since  FVPDS  contains 
unclassified  information.  There  are  a  number  of  cross  references  available  which  make  it 
easy  to  obtain  the  whole  picture  on  a  stated  problem.  The  system  produces  a  number  of 
standard  reports  which  may  be  viewed  on  the  screen  and/or  printed.  Tailored  reports  may 
also  be  produced  for  special  purposes.  Tailoring  may  be  done  by  the  user,  as  can  query,  for 
matting  and  running. 

The  benefits  of  FVPDS,  as  seen  by  TACOM,  are  that  it  will  help  people  at  TACOM 
solve  business  problems  faster  and  at  less  cost  and  will  provide  1%  to  2%  discretionary 
hours  that  coum  be  used  more  creatively  to  benefit  the  command. 
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The  intangible  benefits  that  could  be  derived  from  FVPDS  as  stated  by  TACOM  are  listed 
below: 

Greater  Preparedness 

Greater  Trust  in  TACOM 

More  Confident  Soldiers 

Higher  Quality  Work 

Better  Educated  Workers 

Better  Vehicle  Designs 

Proactive  Influence  on  Fielded  System 

Designed  by  TACOMers  for  TACOMers  and  transportability  to  the  other  MSCs 
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3.0  Status  of  Accomplishments 

■  3.1  Statistical  Process  Control 

As  stated  in  the  discussion,  initiation  of  an  investigation  was  examined  for  the  whole 
process  and  also  by  action  officer.  These  results  are  preliminary  and  now  await  PAD 
management  review  for  a  determination  as  to  what  the  next  step  will  be. 

3.2  Predictive  Analysis  Plan 

The  plan  has  been  released  to  PAD  and  preliminary  analysis  of  the  TACOM 
Fielded  Vehicle  Performance  Data  System  (FVPDS)  completed.  This  task  is  also  waiting 
management  review  and  guidance. 

4.0  Summary 

4.1  The  automation  of  statistical  process  control  on  the  QDR  processing  actions 
appears  to  be  possible  within  the  UNIX  system  as  it  exists.  A  review  of  available 
commercial  software  may  reveal  a  product  that  meets  PAD’s  requirements  and  saves 
programming  time.  For  initiation  of  investigation  a  decision  is  required  as  to  what  the 
acceptable  limits  are  and  these  need  to  be  made  known  to  the  personnel  doing  the  work. 
They  must  also  be  made  aware  of  the  mechanics  of  the  SPC  process  as  to  what  is  being 
measured  and  how  and  the  purpose  for  the  measurement  must  be  fully  understood  and 
agreed  upon  by  all  personnel,  management  and  worker. 

4.2  The  predictive  analysis  plan  presented  to  PAD  outlines  a  zero  base  start  to  achieve 
that  capability.  The  start  of  the  initial  review  of  available  systems  turned  up  the  TACOM 
FVPDS,  which  appears  to  be  a  viable  candidate  for  modification  and  adoption  by  MICOM. 
FVPDS  is  considered  viable  because  it  is  written  around  a  proposed  AMC  standard 
database  management  system  that  will  allow  the  application  of  command  unique  routines 
and  it  uses  several  sources  mentioned  in  the  plan  submitted.  A  closer  look  at  the  TACOM 
product  is  warranted  and  close  coordination  with  Mr.  John  McGowen,  MLC,  876-9018,  the 
MICOM  POC  for  FVPDS  is  required. 
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5.0  Recommendations 

5.1  Since  presentation  of  data  is  a  big  portion  of  the  PAD  requirements  and  the  Sperry 
is  somewhat  limited  in  its  graphics  capability,  it  is  recommended  that  commercial  SPC 
software  packages  be  reviewed  for  a  possible  fit.  At  a  minimum,  PAD  management  must 
set  time  limits  for  those  items  they  desire  to  measure  that  are  not  now  set  by  regulation. 
The  personnel  performing  the  actions  being  measured  must  be  made  aware  of  what  is 
being  measured,  how  it’s  being  measured  and  why  its  being  measured  and  should  be  a  part 
of  the  goal  setting  process. 

5.2  It  is  recommended  that  a  much  more  detailed  look  at  the  TACOM  FVPDS  be 
completed  before  any  decision  on  pursuing  a  unique  MICOM  program  be  made. 
Particular  attention  must  be  paid  to  hardware  requirements  and  MICOM  management 
desires  versus  FVPDS  products. 
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APPENDIX  A 
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ABSTRACT 


This  report  documents  a  proposed  plan  for  the  development  of  a  predictive  analysis 
capability  within  the  MICOM  Product  Assurance  Directorate’s  Customer  Feedback  Office. 
This  plan  is  proposed  by  Hilton  Systems  Incorporated  and  employs  a  sequential 
methodology  common  to  the  development  of  a  Decision  Support  System. 


1.0  PyRPOSE 

The  purpose  of  this  report  is  to  propose  a  plan  for  the  development  of  a  predictive 
analysis  capability  within  the  Customer  Feedback  Office  (CFO)  of  the  MICOM  Product 
Assurance  Directorate  for  government  comment. 


The  objective  of  this  program  is  to  obtain  failure  data  from  as  near  real  time  data 
sources  as  possible,  identify  early  trends  and  predict  failures  based  on  these  trends,  thereby 
precluding  future  deliveries  of  defective  hardware  to  the  field  units.  N 
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2.0  BACKGROUND 


The  1985  Kerwin  Board  II  report  on  the  review  of  the  Army  Materiel  Command’s 
(AMC)  product  assurance  program  recommended  that  within  each  subordinate  command 
AMC  establish  a  closed  loop  process  of  data  feedback  and  analysis,  which  assured  timely 
analysis,  resolution  and  response  to  the  field  as  well  as  provided  information  and  internal 
actions  to  appropriate  functional  organizations.  This  process  would  include  Quality 
Deficiency  Reports  (QDR’s),  Sample  Data  Collection  (SDC),  Equipment  Improvement 
Recommendation  (EIR),  demand  data,  initial  fielding  team  reports  and  test  reports. 

The  commander  of  AMC  initiated  the  recommendation  in  an  August  1986  directive 
that  each  major  subordinate  command  establish  a  Customer  Feedback  Center.  MICOM 
approved  a  provisional  office  in  the  Product  Assurance  Directorate  in  September  1986, 
followed  in  May  1987  with  the  establishment  of  the  present  Customer  Feedback  Office, 
CFO. 


MICOM  R  10-2  in  part  states  that  the  mission  of  the  CFO  is  "to  provide  analyses  of 
all  available  data  on  the  Deficiencies  of  MICOM  managed  equipments  and  to  make 
recommendations,  based  on  the  analyses,  for  appropriate  action."  Further,  function 
number  10  states  that  CFO  will  "perform  proactive/predictive  analyses  of  the  DRS  and 
other  related  data  to  support  the  materiel  life  cycle  phases  of  the  command’s  weapon 
systems." 
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3.0  PROPOSED  PLAN 

The  sequential  methodology  depicted  in  Appendix  A  is  proposed  as  the  plan  to 
accomplish  the  development  of  a  computer  aided  predictive  analysis  capability  within  CFO. 
The  following  subparagraphs  provide  a  description  of  each  major  element  in  the  plan  and 
Appendix  B  provides  a  recommended  timeframe  for  completion  of  the  development. 


3.1  Define  Goals,  Objectives  and  Constraints 

The  following  actions  must  be  taken  to  properly  define  the  desired  system: 

1.  Discussions  with  PAD  managers 

2.  Identify  data  element  needs 

3.  Review  existing  functions,  directives,  studies,  etc. 

4.  Define  limitations  of  PAD  hardware 

5.  Formulate  written  goals  and  objectives. 

The  discussions  with  the  PAD  managers  will  provide  the  direction  and  guidance 
needed  in  formulating  the  program  goals  and  objectives.  Such  questions  as:  What  problem 
are  we  solving  and  What  resources  are  available  to  solve  it?  will  be  partially  answered 
through  these  discussions.  These  discussions  will  also  identify  the  data  elements  that  the 
managers  deem  essential  for  their  decision  making  process. 

Review  of  existing  functions,  directives,  etc.,  will  establish  a  base  from  which  to 
build  and  ensure  that  needed  inputs  and  actions  are  not  overlooked.  It  has  been  learned 
that  experienced  personnel  may  take  undocumented  and/or  unconscious  "short  cuts”  that 
must  be  captured  and  incorporated  into  the  program. 

The  limitations  of  the  existing  PAD  hardware  must  be  determined  so  that  such 
technical  considerations  as  database  size,  organization,  operational  interface  and 
information  display  are  thoroughly  considered.  Overcoming  any  short  falls  can  be 
accomplished  through  descoping  the  program,  use  of  the  MiCOM  information 
management  directorate  systems,  or  placing  additional  equipment  on  procurement  action. 


3 2  Identify  Existing  Data  Sources 

Once  the  data  needed  are  agreed  upon,  existing  army  databases,  programs,  etc.  will 
be  evaluated  to  determine  if  a  ready  source  exists.  The  aim  is  not  to  transfer  all  the 
database  but  to  select  only  those  data  elements  applicable  to  the  MICOM  requirement. 
The  program  developer  must  also  avoid  imposing  additional  reporting  requirements  on 
field  units. 

Some  possible  sources  of  data  include: 

1.  MICOM  -  Provisioning  Master  Record 

•  MLC  Demand  Data  and  Usage  Data 

2.  MRSA  -  The  Army  Maintenance  Management  System  Database 

-  Work  Order  Logistics  File 
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3.  Precido  -  Central  Demand  Database 

4.  Logistics  Center, Ft.  Lee  -  Standard  Army  Maintenance  System 

Level  3 

-  Standard  Army  Ammunition  System 
Level  1 

5.  AMC  -  Standard  Depot  System 

•  Logistics  Intelligence  File 
-  Commodity  Command  Standard  System 

3.3  Evaluate  Existing  Systems/Hardware  and  Software 

Here  again  the  aim  is  not  to  find  a  system  to  incorporate  but  to  capture  and 
implement  any  usable  portions.  Incorporation  of  existing  code  saves  programming  time 
and  thus  saves  money  in  the  development  while  delivering  a  system  sooner. 

The  hardware  in  use  with  the  other  system  must  be  examined  to  ensure  software 
compatibility  with  the  MICOM  hardware. 

There  are  systems  available  at  AVSCOM  and  TACOM  that  will  be  evaluated  with 
these  purposes  in  mind.  Flexibility  of  this  software  is  a  key  element;  it  must  be  capable  of 
being  adapted  to  changes  needed  to  meet  the  MICOM  manager’s  needs. 

Another  aspect  of  evaluating  existing  systems  is  commercial  software.  If  the  job 
desired  mainly  extracts  data  from  a  database  and  produces  reports  based  on  aggregating 
the  data,  the  existing  query  language  may  be  all  that  is  needed.  This  is  based  on  the  notion 
that  "if  I  had  all  the  relevant  information  in  front  of  me  and  presented  in  an  easily 
interpreted  form,  I  could  make  a  decision."  Even  if  the  reports  are  to  be  based  on 
statistical  analysis  of  the  data  rather  than  simple  aggregations,  a  commercial  statistics 
package  (Minitab,  SAS,  SPSS,  etc.)  may  be  used  along  with  the  query  language. 


3.4  Establish  Requirements 

After  all  the  data  and  information  mentioned  above  has  been  collected  and 
evaluated,  the  developer  must  again  meet  with  the  PAD  managers  and  refine  the  goals  and 
objectives  into  a  set  of  system  requirements. 

These  requirements  will  be  used  by  the  programmers  to  develop  a  functional 
description  of  the  system  and  a  set  of  program  specifications.  These  will  be  presented  to 
the  government  for  review,  comment  and  concurrence. 


3.5  Interface  Databases 

The  purpose  of  this  step  is  to  establish  the  method(s)  needed  to  secure  the  required 
data  elements  previously  identified. 
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This  could  be  a  simple  direct  download  into  the  CFO  database  or  so  complicated  as 
*  to  require  the  development  of  a  translation  program  or  as  time  consuming  as  keyboard 
entry  of  hardcopy  information,  the  latter  of  which  is  seen  as  tdtally  unacceptable  for  the 
purpose  proposed. 


3.6  Develop  Analysis  Programs 

The  approach  to  be  used  in  this  step  is  the  standard  develop-use-learn-develop 
methodology  employed  in  decision  support  system  development.  For  this  procedure  the 
overall  program  is  broken  down  into  logical  tasks  rather  than  one  huge  effort.  A  portion  is 
programmed,  given  to  the  user  for  hands-on  evaluation,  his/her  comments  and  suggestions 
and/or  desires  are  noted  and  incorporated  to  the  maximum  extent.  This  process  is  iterated 
until  the  desired  product  is  achieved. 
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APPENDIX  A 
PLAN  OUTLINE 


1.  Define  Goals,  Objectives  and  Constraints 

a.  Discussions  with  PAD  Managers 

b.  Identify  Data  Elements  Needed 

c.  Review  Existing  Functions,  Directives,  Studies,  etc. 

d.  Define  Limitations  of  PAD  Hardware 

e.  Formulate  Written  Goals  and  Objectives 

2.  Identify  Existing  Data  Sources 

a.  MICOM 

b.  MRSA 

c.  Precido 

d.  Logistics  Center,  Ft.  Lee 

e.  AMC 

3.  Evaluate  Existing  Systems/Hardware  and  Software 

a.  AVSCOM 

b.  TACOM 

c.  PAD;  CFO,  Tl,  RA 

4.  Establish  Requirements 

5.  Interface  Databases 

6.  Develop  Analysis  Programs 
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